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SYSTEMS AND METHODS FOR INTERACTIVE NAVIGATION 
AND VISUALIZATION OF MEDICAL IMAGES 
Cross-Referencc to Related Application 
This application claims priority to U.S. Provisional Application No. 60/617»SS9, filed 
on October 9, 20049 which is fiilly incorporated herein by reference. 

Technical Field of the Invention 
The present invention relates generally to systems and methods for aiding in medical 
diagnosis and evaluation of internal organs (e.g., blood vessels, colon, heart, etc,) More 
specifically, the invention relates to systems and methods that support visualization and 
interactive navigation of virtual images of internal organs, and other anatomical components, 
to assist in medical diagnosis and evaluation of internal organs. 

Background 

Various systems and methods have been developed to enable two-dimensional ("2D") 
visualization of human organs and other components by radiologists and physicians for 
diagnosis and formulation of treatment strategies. Such systems and methods include, for 
example, x-ray CT (Computed Tomogr^hy), MRI (Magnetic Resonance Imaging), 
ultrasound, PET (Positron Emission Tomography) and SPECT (Single Photon Emission 
Computed Tomography). 

Radiologists and other specialists have historically been trained to analyze image scan 
data consisting of two-dimensional slices. Three-Dimensional (3D) images can be derived 
from a series of 2D views taken from different angles or positions. These views are 
sometimes referred to as "slices" of the actual three-dimensional volume. Experienced 
radiologists and similariy trained personnel can often mentally correlate a series of 2D 
images derived from tiiese data slices to obtain usefiil 3D information. However, while stacks 
of such slices may be useful for analysis, they do not provide an efiBcient or intuitive means 
to examine and evaluate interior regions of organs as tortuous and complex as a colons or 
arteries. For example, vAim imaging blood vessels, 2D cross-sections merely show slices 
through vessels, making it difficult to diagnose stoiosis or other abnormalities. Moreover, 
with 2D images of colons, it can be difficult to distinguish colonic polyps from residual stool 
or normal anatomical colonic features such as haustral folds. 

In this regard various techniques have been and are continually being developed, to 
enable 3D rendering and visualization of medical image datasets, wherein the entire volume 
or portion an imaged organ can be viewed in a 3D virtual space. For instance, 3D virtual 
endoscopy applications include methods for rendering endoscopic views of hollow organs 
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(such as a colon or blood vessels) and allowing a user to navigate the 3D virtual image space 
of an imaged colon or blood vessel, for example, by flying through the organ lumen while 
viewing the inner lumen walls. While navigation and exploration the 3D image space of a 
virtual organ can provide an efficient or intuitive means to examine and evaluate interior 
regions of organs, a user can become confused and lose his/her sense of direction and 
orientation while navigating in virtual space. In this regard, it is desirable to implement 
mediods for assisting user navigation in a complex virtual image space. 

Summary of the Invention 
In general, exemplary embodiments of the invention include systems and methods for 
visualization and interactive navigation of virtual images of internal organs to assist in 
medical diagnosis and evaluation of internal organs. In one exemplary embodiment, an image 
data processing system includes an image rendering system for rendering multi-dhnensional 
views of an imaged object fiiom an image dataset of the imaged object, a graphical display 
system for displaying an image of a rendmd view according to specified visualization 
parameters, an interactive navigation system which monitors a user's navigation through a 
virtual image space of a displayed image and which provides user navigation assistance in the 
form of tactile feedback by a navigation control unit operated by Ae user, upon an occurrence 
of a predefined navigation event 

In one exemplary embodiment, force feedback is applied to a steering control unit of 
the navigation control device to guide the user's flight path in a direction along a 
predetermined flight path. The predetermined flight path may be a centerline through a lumen 
of a hollow organ (such as a colon or blood vessel). The predefined event is based on a 
distance of the virtual camera from the predetermined flight path. The magnitude of the force 
feedback applied to the steering control unit may vary based on a measure of a distance of the 
virtual camera fix>m the predetermined flight path. 

In another exemplary embodiment of the invention force feedback is applied to a 
steering control unit of the navigation control device to guide the user's flight path in a 
direction away torn an anatomical object to avoid collision with the object. For virtual 
endoscopy applications^ the anatomical object is a virtual lumen inner wall. The predefined 
event is based on a distance of the virtual camera to the lumen inner wall. The magnitude of 
the force feedback applied to the steering control unit can vary based on a measure of the 
distance of the virtual camera to the anatomical object (e.g., lumen wall). A force feedback 
may also be applied to a flight speed control unit of the navigation control device to reduce or 
stop the user's fli^t path to avoid collision with the anatomical object. 
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In another exemplaiy embodiment of the invention, force feedback can be applied to a 
flight speed control unit of the navigation control device to reduce a flight speed and allow 
the user to review a region of interest that the user may have missed. For example, the 
predefined event can be based on a tagged region of interest entering a field of view of a 
virtual camera A foroe feedback can be S4)plied to a steering control unit to guide user's 
flight path in a direction toward the tagged region of interest. 

In anoth^ exemplary embodiment of the invention, interactive navigation assistance 
is provided by automatically modulating a user's flight speed upon the occurrence of a 
triggering event while navigating through a virtual image space such that a perceived flight 
speed remains substantially constant as tiiie user navigates through the virtual image space. 
For instance* in virtual mdoscopy applications, the triggering event may be based on 
threshold measures of increasing^decreasing lumen width while navigating along a lumen 
centerline, or threshold distance measures with regard to the distance between a virtual 
camera (view point) and a lumen wall. The actual flight speed is gradually reduced or 
increased as the distance between tfie virtual camera and lumen wall decreases or increases, 
respectively, while navigating along a flig^it path. 

In one exemplary embodiment, flight speed is automatically modulated by oveniding 
an input event generated by user operation of a flight speed control unit. In another 
embodiment, flight speed is automatically modulated by providing force feedback to a flight 
speed control unit operated by a user to automatically control the flight speed control unit. 

These and other exemplary embodiments, aspects, features and advantages of the 
present invention will become apparent from the following detailed description of preferred 
embodiments, which is to be read in coimection with the accompanymg drawings. 

Brief Description of the Drawings 

FIG. 1 is a diagram of an imaging system according to an mibodiment of the 
invention. 

FIG. 2 is a flow diagram illustrating a method for providing interactive navigation 
according to exemplary embodiments of the invention. 

FIG. 3A illustrates an exemplary 3D overview of an imaged colon having a specified 
flight path through the colon lumen. 

FIG. 3B schematically illustrates a method for providing force feedback to control the 
direction of a user flight path, according to an exemplary embodiment of the invention. 

FIG. 4 is a flow diagram illustrating a method for automatically modulating fli^t 
speed during user navigation to mainUun a constant perceived flight speed, according to an 
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exemplary embodiment of the invention. 

FIG. 5 is a flow diagram illustrating a method for fusing and/or overlaying secondary 
information over a primary 2D/3D view according to an exemplary embodiment of the 
invention. 

FIG. 6 illustrates a method for overlaying secondary information in a primary view 
according to an exemplary embodiment of the invention. 

FIG. 7 is an exemplary filet view of a colon sur&ce according to an exemplary 
embodiment of the invention. 

Detailed Description of Exemplary Embodiments 

Exemplary systems and methods for providing visualization and interactive 
navigation of virtual images of internal organs, and other anatomical components, will now 
be discussed in further detail. It is to be understood that the systems and methods described 
herein may be implemented in various forms of hardware, software, fimware, special 
purpose processors, or a combination thereof. For example, the methods described herein 
may be implemented in software as program instructions that are tangibly embodied on one 
or more program storage devices (e.g., magnetic floppy disk, RAM, CD ROM, DVD ROM, 
ROM and flash memory), and executable by any device or machine comprising suitable 
architecture. It is to be fiutiier understood diat since the constituent q^stem modules and , 
method steps depicted in Ae accompanying Figures may be Implemented in software, the 
actual connection between the system components (or the flow of the process steps) may 
differ depending upon the manner in which tibe present invention is programmed. Given the 
teachings herein, one of ordinary skill in the related art will be able to contemplate these and 
similar implementations or configurations of the present invention. 

FIG. 1 is a diagram of an ima^ng system (100) according to an embodiment of the 
present invention. The imaging system (100) comprises an image acquishion device tiiat 
generates 2D image datasets (101) which can be formatted in DICOM format by a DICOM 
processing system (1 02). For instance, the 2D image dataset (101) may comprise a CT 
(Computed Tomography) dataset (e.g., Electron-Beam Computed Tomography (EBCT), 
Multi-Slice Computed Tomography ^CT), etc.), an MRI (Magnetic Resonance Ima^ng) 
dataset, an ultrasound dataset, a PET (Positron Tomography) dataset, an X-ray dataset or a 
SPECT (Single Photon Emission Computed Tomography) dataset. A DICOM server (1 03) 
provides an interfiice to the DICOM system (102) and receives and process the DICOM- 
formatted datasets received fiom the various medical image scanners. The server (103) may 
comprise software for converting the 2D DICOM-formatted datasets to a volume dataset 
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(I03a). The DICOM server (103) can be configured to, e.g., continuously monitor a 
hospital network (104) and seamlessly accept patient studies automatically into a system 
database the moment such studies are ^'pushed" fix)m an imaging device. 

The imaging system (100) further comprises an imaging tool (lOS) that executes on a 
computer system. The imaging tool (105) comprises a repository (106) for storing image 
datasets and related meta information, an interactive navigation module (107), a segmentation 
module (108), a multi*modal image fusion module (109), an automated diagnosis module 
(1 10), an image rendering module (111), a user inter&ce module (1 12), a database of 
configuration data (1 13), and a feedback control system (1 14). A user interacts with the 
ima^ng tool (lOS) using one or more of a plurality of I/O devices including an interactive 
navigation control device (1 1 5) and/or a screen, keyboard, mouse, etc. (1 1 6). As explained 
below, the feedback control system (1 1 4) and navigation control device (1 1 S) operate to 
provide one or more forms of tactile feedback to a user when navigatmg through a virtual 
image space to provide interactive navigation assistance. 

The imaging tool (lOS) may be a heterogeneous image processing tool diat includes 
methods for processing and rendering image data for various types of anatomical organs, or 
the imaging tool (105) may implement methods that are specifically designed and optimized 
for processing and rending image data of a particular organs. The imaging tool (1 05) can 
access tiie DICOM server (103) over the networic (104) and obtain 2D/30 DICOM formatted 
image datasets that are stored in the local repository (106) for further processing. 

The user interfiu^e module (1 12) implements methods to process user input events 
(mouse clicks* keyboard inputs, etc.) for purposes of executing various image processing and 
rendering functions supported by the imaging tool (105) as well as.setting/selecting/changing 
system parameters (e.g., visualization parameters), which are stored as configuration data in 
the database (1 13). The GUI module (1 12) displays 2D/3D imag^ from 2D/3D views that 
are rendered by the rendering module (1 1 1). 

The rendering module (1 1 1) implements one or more 2Dy3D image rendering 
methods for generating various types of 2D and 3D views based on user specified and or 
de&ult visualization parameters. Preferably, the 2D/3D rendering methods support functions 
such support real-time rendering of opaque/transparent endoluminal and exterior views, 
rendering of view with superimposed or overlaid images/infomfiation» (e.g., superimposed 
centerlines in colonic endoluminal views, user adjustment of window/level parameters 
(contrast/brightness), assignment of colors and opacities to image data (based on default or 
user modified transfer functions which map ranges of intensity or voxel values to different 
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colors and opacities), user interaction with and manipulation of rendered views (e.g., 
scrolling, taking measurements, panning zooming, etc.). Hie rendering module (1 1 1) 
generates 2D and 3D views of an image dataset stored in the repository database (106) based 
on the viewpoint and direction parameters (i.e., current viewing geometry used for 3D 
rendering) received fix>m the GUI module (1 12). The repository (106) may include 3D 
models of original CT volume datasets and/or tagged volumes. A tagged volume is a 
volumetric dataset comprising a volume of segmentation tags that identify which voxels are 
assigned to which segmented components, and/or tags corresponding other types of 
information which can be used to render virtual images. When rendering an image, the 
rendering module (1 1 1) can overlay an original volume dataset with a tagged volume, for 
example. 

The segmentation module (108) implements one or more known automated or semi- 
automated methods segmenting features or anatomies of interest by reference to known or 
anticipated image charactBristics» such as edges, identifiable structures, boundaries, changes 
or transitions in colors or intensities, changes or transitions in spectrographic information, 
etc. The segmentation module (108) comprises methods tiiat enable user interactive 
segmentation for classifying and labeling medical volumetric data. The segmentation module 
(108) comprises functions that allow the user to create, visualize and adjust the segmentation 
of any region within ortiiogonal, oblique, curved MPR slice image and 3D rendered ims^es. 
The segmentation module (108) is interoperable with annotation methods to provide various 
measurements such as widtii, height, length volume, average, max, std deviation, etc of a 
segmented region. Various types of segmentation metiiods that can be implemented are well 
known to those of ordinary skill in the art, and a detmled discussion thereof is not necessary 
and beyond the scope of the claimed inventions. 

The automated diagnosis module (1 10) implements methods for processing image . 
data to detect, evaluate and/or diagnose or ottierwise classify ehnomal anatomical structures 
such as colonic polyps, aneurisms or lung nodules. Various types of methods that can be 
implemented for automated diagnosis/classification are well known to tiiose of ordinary skill 
in the art, and a detailed discussion thereof is not necessaiy and beyond the scope of tiie 
claimed inventions. 

The multi-modal image fusion module (109) implements metiiods for fusing 
(registering) image data of a given anatomy that is acquired fix>m two or more imaging 
modalities. As explained below with reference to FIG. S-7, the multi-modal image fiision 
module (109) implements methods for combining different modes of data in a manner that 
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allows the rendering module (111) to generate 2D/3D views using different modes of data to 
thereby enhance the ability to evaluate imaged objects. 

The interactive navigation module (107) implements methods that provide interactive 
user navigation assistance to a user when navigating through a virtual imi^e space. For 
5 example, as explained in further detail below, methods are employed to monitor a user's 

navigation (flight path and/or flight speed, for example) though a virtual image space (2D or 
3D space) and provide some form of tactile feedback to the user (via the navigation control 
device (1 1 S)) upon the occurrence of one or more predefined events. As explained below, 
tactile feedback is provided for puiposes of guiding or otherwise assisting the user^s 

10 exploration and viewing of the virtual image space. 

In accordance widi an exemplary embodiment of the invention, navigation through 
virtual image space is based on a model in which a Virtual camera** travels through s virtual 
space witfi a view direction or "lens" pointing in the direction of the current flight path. 
Various methods have been developed to provide camera control in the context of navigation 

IS within a virtual environment For instance, U.S. Patent Application Seriid No. 10/496,430, 
entitled "Re^stration of Scanning Data Acquired fiom Dififerent Patient Positions" (which is 
commonly assigned and fully incorporated herein by reference) describes methods for 
generating a 3D virtual image of an object such as a human organ using volume visualization 
techniques, as well as methods for exploring the 3D virtual image space using a guided 

20 navigation system. The navigation system allows a user to travel along a predefined or 
dynamically computed flight path through the virtual image space, and to adjust both the 
position and viewing angle to a particular portion of Interest in the image away from such 
predefined path in order to view regions of interest (id^tify polyps, cysts or other abnormal 
features in an organ). The camera model provides a virtual camera that can be fiilly operated 

25 with six degrees of freedom (3 degrees movement in horizontal, vertical, and depth directions 
(x,y,z) and 3 degrees of angular rotations) in a vutual environment to thereby allow the 
camera to move and scan all sides and angles of a virtual environment 

In accordance with one embodiment of tfie invention, the navigation control device 
(1 IS) can be operated by a user to control and manipulate the orientation/direction and flight 

30 speed of the "virtual camera". For instance, in one exemplary embodiment of the invention, 
the navigation control device (11 S) can be a handheld device having a joystick that can be 
manipulated to change the direction/orientation of the virtual camera in the virtual space. 
More specifically, in one exemplary embodiment, the joystick can provide two-axis (x/y) 
control , where the piteh of the virtual camera can be assigned to the y-axis (and controlled by 

7 
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moving the joystick in a direction up and down) and where the heading of the virtual camera 
can be assigned to the x-axis (and controlled by moving the joystick in a direction left and 
right). The navigation control device (115) may further include an acceleration button or 
pedal, for instance, that a user can press or othenvise actuate (with varying degrees) to 
control the velocity or flight speed of the virtual camera along a user-desired flight path 
directed by user manipulation of the joystick. 

When free flying through a 3D space (such as a within a colon), a user can lose a 
sense of direction and orientation or otherwise navigate at some flight speed along some 
flight path that causes the user to inadvertently pass some region of interest in the virtual 
image space tiie user may have found to be of particular interest for careful examination. Iii 
this regard, the navigation control device (1 IS) can be adapted to provide some form of 
tactile feedback to the user (while operating the control device (I IS) in response to feedback 
control signals output from the feedback controller (1 14). The feedback controller (1 14) can 
generate feedback control signals under command from the interactive navigation module 
(107) upon the occurrence of one or more pre-specified conditions (as described below) for 
triggering user^assisted navigation. The navigation control device (115) provides appropriate 
tactile feedback to the user in response to the generated feedback control signals to provide 
the appropriate user navigation assistance. 

FIG. 2 is a flow diagram illustrating metiiods for providing interactive navigation 
according to exemplary embodiments of the invention. As an initial step, the imaging system 
will obtain and render an image dataset of an imaged object (step 20). For instance, in a 
virtual colonoscopy ^plication, the image dataset may comprise a 3D volume of CT data of 
an imaged colon. In one exemplary embodiment of the invention, to support some type(s) of 
user-assisted navigation, the imaging system will provide a specified flight path throu^ the 
virtual image space of the image dataset (step 21). In one exmiplary embodiment of the 
invention, a fly-patii througti a virtual organ, such as a colon lumen, is generated. For 
instance, FIG. 3A illustrates a 3D overview of an imaged colon (30) having a specified fli^t 
patii through the colon lum^. In the exemplary embodiment, the specified flight patii is a 
center line C tiiat is computed inside the colon lumen, and such path can be traversed for 
navigating through the colon at the center of the colon. The centeriine C can be computed 
using known metiiods such as those disclosed in U.S. Pat. No. 5,971,767 entitied "System 
and Method for Performing a Three-Dimenstonal Virtual Examination**, which is 
incorporated by reference herein in its entirety. 

It is to be understood that the use of a pre-specified flight path is optional. As will be 
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explained below, a pie-specified flight path can be implemented to support one or more 
forms of interactive user navigation assistance. In other exemplary embodiments of the 
Invention, interactive user navigation assistance can be provided without use of a pre- 
specified flight path. 

s The system will process user input from a navigation control device that is 

manipulated by the user to direct the movement and orientation of a virtual camera along a 
given flight path (step 22). In one exemplary embodiment of the invention, the user can 
traverse the pre-specified flight path (e.g., colon centerline C) or freely navigate along a user 
selected flight path that diverges from the pre-specified flight padi. In particular, the user can 

10 navigate throu^ the virtual space using the pre-specified fli^t path, whereby the virtual 
camera automatically travels along die pre-specified fli^t path with the user being able to 
control the direction and speed along the pre-specified flight path by manipulating the input 
control device. In addition, the user can freely navigate throu^ the virtual space away from 
the pre-specified flight path by manipulating the control device appropriately. 

15 As the user navigates through the virtual space, the system will render and display a 

view of the imaged object from the view point of the virtual camera in the direction of the 
given flight path (specified or user-selected path) (step 23). For 3D visualization and 
navigation, any one of well-known techniques for rendering and displaying images in real- 
time may be implemented, the details of which are not necessary and outside the scope of this 

20 invention. As the user navigates through the virtual space, the system will provide interactive 
navigation assistance by automatically providing tactile feedback to the user via the input 
control device upon the occurrence of some predetermined condition/event (step 24). The 
type of tactile feedback can vary depending on the 

For instance, in one exemplary embodiment, the interactive navigation module (107) 

25 can track a user's flight path in a 3D virtual image space within an organ lumen (e.g., colon) 
and provide force feedback to the input control device to guide the user's path along or in 
proximity to the pre-specified flight path (e.g., centerline of a colon lumen). In this regard, a 
feedback controller (1 14) can generate control signals that are applied to the control device 
(115) to generate the force feedback to the joystick manipulated by the user as a way of 

30 guiding die user's free flight in the direction of the pre-specified flight path. By way of 

example, FIG. 3B schematically illustrates a method for providing force feedback to control 
the direction of the flight path. FIG. 3B illustrates an exemplary virtual space (colon lumen) 
having a pre-specified path (e.g., colon centerline C) and a virtual camera at position P and a 
user selected direction D. The navigation control device (1 1 5) can be controlled to apply an 
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appropriate feedback force to the joystick to help guide the user's path in the direction Dl in 
the vicinity of the pre-specified path C. 

In the exemplary embodiment of FIG. 3B, a corrective force that must be applied to 
the input device to yield tfie direction D; can be computed using any suitable metric. For 
instance, the magnitude of the applied feedback force can be a function of the current 
distance between the virtual camera and the pre-specified path, whereby the feedback force 
increases the further away the virtual camera is from the pre-computed path. On the other 
hand, \^en the virtual camera is close to the pre-specified path, a gentle feedback foixie can 
be applied to the joystick guide the user along the pre-specified path. TTiis form of tactile 
feedback enhances the user*s ability to freely manipulate a camera in 3D space while staying 
true to a pre-computed optimal path. The user can override or otherwise disregard such 
feedback by forcibly manipulating the joystick as desired. The user may release the joystick 
and allow the force feedback to automatically manipulate the joystick and thus, allow the 
navigation system to essmtially steer the virtual camera m the in the appropriate direction. 

In another exemplary embodiment, the interactive navigation module (107) could 
provide firee-flight guided navigation assistance without reference to a pre-specified flight 
pa& For instance, went navigating tiirough a organ lumen, force feedback can be applied to 
the joystick in a manner similar to that described above when the virtual camera moves to ' 
close the lumen wall to steer the virtual camera away from the lumen wall and avoid a 
collision. In addition, force feedback can be applied to the flight control button pedal to slow 
down or otherwise stop the movement of die virtual camera to avoid a collision with tiie 
lumen wall. The force feedback can be applied to botii the joystick and flight speed control 
pedal as a means to slow the flight speed of the virtual camera and have time to steer away 
from, and avoid collision witii, the lumen wall. 

In another exemplary embodiment of tiie invention, tactile feedback can be in tiie 
form of a feedback force applied to the flight speed control unit (e.g., pedal, button, or 
throttle slider control, etc.) as a means to control tiie flight speed for otfier purposes (otiier 
than avoiding collision witfi the lumen wall).. For instance, as a user is traveling in virtual 
space along a given patii (user selected or pre-specified patii), tiie system can apply a 
feedback force to die speed control pedal/button as a means of indicating to tiie user tiiat tiie 
user should slow down or stop to review a particular region of interest For instance, tfie 
image data may include CAD marks or tags (e.g., results fiom computer automated diction, 
segmentation, diagnosis, etc.) associated widi tiie image data, which were generated during 
previous CAD processing to indicate regions of interest tiiat are deemed to have potential 
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abnormalities or actual diagnosed conditions (e.g., polyp on colon wall). However, 
depending on various fectors such as the particular view point in the virtual image space, the 
user-selected flight path, the flight speed, etc., the user may inadvertently pass or otherwise 
miss a particular mariced or tagged region of interest in the virtual image that requires a 
careful examination. In this instance, the system can generate control signals to the 
navigation control device to provide force feedback on the flight speed control button/pedal 
as a way of indicating to the user or otherwise forcing the user to reduce the flight speed or 
stop. 

It is to be appreciated that other forms of tactile feedback may be implemented to 
provide interactive navigation assistance, and that die present invention is not limited to force 
feedback. For instance, the input control device can provide tactile feedback in the form of 
vibration. In this instance, the vibration can provide an indication to the user that a current 
region of interest should more carefully reviewed. More specifically, by way of example, 
while navigation in virtual image space, when the virtual camera approaches a marked or 
tagged region of interest, the a combination of force feedback and vibration feedback can be 
applied, whereby the force feedback is applied to the flight speed control button and the 
control device vibrates, to provide an indication to die user that some potential region of 
interest is within the current field of view in proximity to the virtual camera. In another 
embodiment, force feedback can further be applied to the joystick as a means for guiding the 
user to steer the virtual camera in die direction of the potential region of interest 

It is to be appreciated that the types of tactile feedback and the manner in which the 
tactile feedback is implemented to for navigation assistance will vary depending on die 
application and type of control device used. It is to be understood diat die above 
embodiment for tactile feedback are merely exemplary, and that based on the teachings 
herein, one of ordinary skill in the art can readily envision other forms of tactile feedback (or 
even visual or auditory feedback) and applications diereof for providing user navigation 
assistance. 

In anodier exemplary embodiment of die invention, die interactive navigation system 
implements methods for providing automated flight speech modulation to control flight speed 
during user navigation throu^ a virtual space. For instance, when performing a diagnostic 
examination of colon lumen using a 3D endoluminal flight, die examiner must be able to 
effectively and accurately process die information diat is presented during flight. In addition 
to odier factors, die flight speed (or flight velocity) will determine how much and how well 
information is being presented. As such, flight speed can affect how quickly the user can 
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accurately examine the virtual views. More specifically, while navigating at a constant actual 
flight speed (as measured in millimeters/second) the flight speed as perceived by the user will 
vary depending on the distance fix)m the viewpoint to the nearest point on die colon lumen 
surface. 

For example, when navigating through a region of the colon lumen having a gradually 
decreasing or acute decrease in lumen width (i.e., less insufflation), although the user may be 
navigating at a constant speed, there will be a gradual increase or abrupt increase in the 
perceived flight speed by virtue of die viewpoint becoming closer to the colon walls. 
Moreover, when navigating through a region of the colon lumen having a gradually 
increasing or acute increase in lumen width (i.e., more insufflation), although the user may be 
navigating at a constant speed, there will be a gradual decrease or abrupt decrease in the 
perceived flight speed by virtue of the viewpoint becoming further from the colon walls. 

Therefore, as a user is flying through an organ lumen (e.g., colon, blood vessel, etc.), 
die perceived changes in flight speed through areas of varying lumen width can be very 
distracting to the user. In particular, ^en the perceived flight speed increases due to 
decreased lumen width or when the user's flight path approaches the organ wall, it become 
more difiBcult for die user to focus on a particular areas on the lumen wall» because of the 
perception of increased flight speed. Thus, it is desirable to automatically maintain die 
perceived flight speed as constant as possible, without the user having to manually contiol.the 
actual flight speed via the control device. 

FIG. 4 is a flow diagram illustrating a method for automatically modulating fli^t 
speed during user navigation to maintain a constant perceived flight speed. When 
commencing a navigation session, a user can optionally select a function for fli^t speed 
modulation. When the system receives the user request for automated flight speed 
modulation (step 40), the system will specify one or more predetermined events for triggering 
flight speed modulation (step 41). As a user is navigating along a flight path through a 
virtual image space at some constant flight speed (step 42), the system will monitor such 
navigation session for occurrence of a triggering event (step 43). When a triggering event 
occurs (affirmative determination in step 43), the system will automatically modulate the 
actual flight speed such that the user's perceivable flight speed is maintained constant (step 
44). For example, the perceivable fli^t speed is similar to the constant flight speed. In this 
manner, die user can travel at some desirable constant speed, widiout be subject to distracting 
changes in perceived flight speed that can occur under certain circumstances. In one 
exemplary embodiment, automated flight speed modulation can be employed by overriding 
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the user input generated by the user manipulation of a flight speed control unit In another 
exemplary embodiment, automated flight speed modulation can be employed by providing 
force feedback to the flight speed control unit to control the speed using the actual flight 
speed control unit In this manner, tiie user can override the automated flight speed 
modulation, for example, by forcibly manipulating the speed control unit despite the feedback 
force. 

The method depicted in FIG. 4 is a high-level description of a method, which can be 
embodied in various manners depending on the navigation application and type of organ 
being virtually examined. For illustrative purposes, methods for automated flight speed 
modulation according to exemplary embodiments of the invention will be described with 
reference to navigating through an organ lumen and in particular, an endoluminal flight 
through a colon, but it is to be understood that the scope of the invention is not limited to 
such exemplary embodiments. In the context of virtual colonoscopy applications, the 
triggering events can be threshold measures that are based some combination of flight speed 
and distance of view point to the closest point on the lumen wall or some combination of 
flight speed and the lumen width, for example. 

More specifically, by way of example, for virtual colonoscopy applications whei« 
navigation is limited to travel along a specified centerline flight path, for example, the system 
can specify a range of lumen widths having a lower and upper threshold lumen width, 
wherein flight speed modulation is performed when a region in the virtual colon lumen has a 
lumen width outside the tfu^hold range (i.e., the lumen width is less than the lower threshold 
or greater than the upper threshold). In this instance, a triggering event occurs when the user 
navigates to a region of the colon witfiin the current field of view having a lumen width that 
is outside the threshold range. While flying through regions of the colon lumen having 
widdis greater than the upper threshold, the decrease is perceived flight speech may not be 
too distracting to tfie user and as such, modulating may not be implemented. However, for 
lumen widths less than the lower threshold, the increased in perceived flight speed is 
undesirable, so modulation of the flight speed in such circumstance is desirable. It is to be 
appreciated that the threshold range of lumen widths can be dynamically varied depending on 
the user's current flight speed. For instance, at higher flight speeds, the range may be 
increased, while the range may be decreased for lower flight speeds. 

Any suitable metric may be used for modulating tiie flight speed. In one exemplary 
embodiment, when traveling to regions of decreased lumen width, the actual flight speed is 
modulated using some metric based on the lower threshold width. For instance, a 
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neighborhood sample of lumen widths are taken and averaged. The resulting change in 
velocity can be dynamically computed as some percentage of the averaged lumen width 
according to some specified metric. This metric is specified to avoid abrupt changes in flight 
speed due to sharp changes in lumen width (e.g., narrow protruding object). The result is a 
gradual reduction of the actual flight speed as the user's field of view encounters and passes 
thru areas of decreased lumen width resulting in little or no perceivable increase in flight 
speed. In this manner, the user can travel along the centeriine of the colon lumen at a 
constant speed, while being able to examiner regions of smaller lumen width without having 
to manually reduce the flight speed. 

In another exemplary embodiment of the invention, for virtual colonoscopy 
applications where navigation is not limited to travel along a specified centeriine flight path» 
for example, the system can specify a minimum distance threshold, wherein flight speed 
modulation is performed when the distance between the viewpoint and a closest point on the 
lumen wall fiilis below the minimum distance threshold. In this instance, a triggering event 
occurs when the user navigates at some constant flight speed and moves the view point close 
to the lumen wall such that thm is a perceived increase in fli^t speed with respect to 
proximate regions of the Ixmsxi wall. In such instance, modulation of the flight speed is 
desirable to avoid an increase in the perceived flight speed. It is to be appreciated that the 
minimum distance threshold range can be dynamically varied depending on the user's current 
flight speed. For instance, at higher flight speeds, the distance threshold can be increased, 
while the distance tfireshold may be decreased for lower flight speeds. 

Any suitable metric may be used for modulating the flight speed. In one exemplary 
embodiment, when navigating close to a lumen wall, the actual flight speed is modulated 
using some metric based on the minimum distance threshold. For instance, a neighborhood 
sample of distance measures can be determined and averaged. The resulting change in 
velocity can be dynamically computed as some percentage of the averaged distance according 
to some specified metric. This metric is specified to avoid abrupt changes in flight speed 
when tiie measure distance to the closest point on the lumen wall is the result of some narrow 
or sharp protrusion or small object on the wall. The result is a gradual reduction of the actual 
flight speed as the user's field of view encounters and passes thru areas of decreased lumen 
width resulting in little or no perceivable increase in flight speed. In this maimer, the user 
can fieely navigate along a desired path through the colon at a constant speed, while being- 
able to closely examine regions of the colon wall without having to manually reduce the 
flight speed. 
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In another exemplaiy embodiment of the invention, as noted above, automated flight 
speed modulation can be implemented in a manner such that a force feedback is {^plied to 
the fli^t speed control unit to reduce or mcrease the flight speed by automated operation of 
the flight speed control unit. The magnitude of the applied force can be correlated to the 
amount of increase or decrease in the actual flight speed needed to maintain a constant 
perceived speed. Again, the user can override the feedback by forcible manipulating the 
speed control unit as desired. 

In other exemplary embodiments of the invention, automated flight speed modulation 
can be implemented And proximity to CAD findings and proximity to features previously 
discovered by the same or other users and proximity to portions of the environment that were 
not previously exampled fully (what we call missed regions), for example. Other possibilities 
include pointing the view direction toward features of interest (CAD findings, bookmarks of 
other users) or in the direction of missed regions. 

In other exemplaiy embodiments of tiie invention, other types of triggering events can 
be defined that initiate other types of automated interactive navigation assistance functions. 
For instance, during a user*s navigation in a virtual inuge space (e.g., 3D endolummal flight) 
the field of view (FOV), which is typically given in degrees fiom left to right and top to 
bottom of image, can be automatically and temporarily increased to aid die user in Wsualizing 
regions of the virtual image space that would otherwise have remamed unseen. The FOV can 
be automatically increased, for instance, while the user is navigating along a path where an 
unseen marked/tagged region of interest is in close proximity such that increasing the FOV 
would reveal such region. Further, during a user's navigation in a virtual image space (e.g., 
3D endoluminal flight) the view direction (along tiie fli^ patfi) can be automatically and 
temporarily modified by ovmiding the user-specified flight path to aid the user in 
visualizing regions of the virtual ima^ space that would ttiat would otherwise have remained 
unseen. For example, the system can automatically steer tiie virtual camera in a direction of 
an unseen marked/tagged region of interest to reveal such region to tfie user. These 
functions can be combined where the system automatically stops the flight, steers the 
viewpoint in the appropriate direction and enlaiges the FOV, to thereby present some region 
of interest to the user, which the user may have missed or passed by wile free-flight 
navigating. 

These automated fimctions can be triggered upon the occurrence of certain events, 
such as based on some distance measure and proximity of the user's current viewpoint to 
tagged legions in the virtual space (e.g., automatically tagged regions based on CAD results 
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(segmentation, detection, diagnosis, etc.) and/or regions in the virtual image space that were 
manually tagged/rnariced by one or more previous users during navi^tion), or unmarked 
regions that deemed to have been missed or unexplored, etc. 

These functions may or may not be implemented in conjunction widi some form of 
feedback (tactile, auditory, visual). When a user's free flight navigation is temporarily 
overridden and automatically modified by the system, some form of feedback would be 
useful to provide some indication to the user of the event In fact, the tactile feedback 
navigation assistance embodiments described above with reference to FIG. 2, for example, 
can be automated functions that are provided without tactile feedback, by simply overriding 
the user's navigation and automatically temporarily controlling the flight speed and flight to 
provide navigation assistance. 

In another exemplary mibodiment of the invention, user navigation and examination 
of a virtual image is supported by implementing methods for rendering images that 
incorporate multi-modal data. For instance, FIG. S is a high-level flow diagram illustrating a 
method for fusing and/or overlaying secondary information over a primary 2D/3D view. In 
one exemplary embodiment, FIG. S illustrates an exemplary mode of operation if the multi- 
modal image fusion module (109) of FIG. 1. An initial step includes generating a primary 
view of an imaged object using image data having a first imaging modality (step SO). For 
instance, in one exemplary embodiment, die image data may be CT data associated with an 
imaged heart, colon, etc. The primary view may be any known view format including, e.g., a 
filet view (as described below), an overview, an endoluminal view, 2D multi-planar 
reformatted (MPR) view (either in an axis orthogonal to the original image plane or in any 
axis), a curved MPR view (where all the scan lines are parallel to an arbitrary line and cut 
through a 30 curve), a double-oblique MPR view, or 3D views using any projection scheme 
such as perspective, orthogonal, maximum intensity projection (MIP), minimum intensity 
projection, integral (sunmiation), or any other non-standard 2D or 3D projection. 

A next step includes obtaining secondary data associated with image data that is used 
for generating the primary view (step 51). The secondary data is combined with associated 
image data in one or more regions of the primary view (S2). An image of the primary view is 
displayed such that those regions of the primary view having the combined secondary 
information are visibly differentiated bom other regions of the primary view (step S3). 

In one exemplary embodiment, the secondary data includes another image data set of 
the image object which is acquired using a second imaging modality, different fix>m the first 
imaging modality. For instance, an image data for a given organ under consideration can be 
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acquired using multiple modalities (e.g., CT, MRI, PET, ultrasound, etc.) and virtual images 
of the organ can be rendered using image data from two or more image modalities in a 
manner that enhances the diagnostic value. In this exemplary embodiment, the anatomical 
image data from different modalities are first processed using a fiision process (or registration 
process) which aligns or otherwise matches corresponding image data and features in the 
difiTerent modality image datasets. This process can be performed using any suitable 
registration method known in the art. 

Once. the image datasets are fiised, a primary view can be rendered using image data 
from a first modality and tfien one or more desired regions of the primary view can be 
overlaid with image data from a second modality using one or more blending methods 
according to exemplaiy embodiments of the invention. For instance, in one exemplary 
mbodimen^ the overlay of infonnation can be derived by selective blending the secondaiy 
information with the primary information using a blending metric, e.g., a metric based on a 
weighted average of the two color images of the different modalities. I another embodiment, 
the secondary data can be overlaid on the primary view by a selective (data sensitive) 
combination of the images (e.g^ the overlaid image is displayed with color and opacity). 

It is to be appreciated that overlaying information from a second image modality on a 
primary image modality can help identify and distinguish abnormal and normal anatomical 
structures (e.g., polyps, stool, and folds in a colon image). For instance. Positron Emission 
Tomography (PET) scanners register the amount of chemical uptake of radioactive tracers 
that are injected into the patient These tracers move to the sites of increased metabolic 
activity and regions of the PET image in which such tracers are extremely concentrated as 
identified as potential cancer sites. Altfiough the information fit>m a PET scan is not very 
detailed (it has a relatively low spatial resolution compared to CT), PET data can be 
extremely helpful when overlaid or embedded over CT or other data using techniques 
described above. The advantage of the overiay of secondaiy information is tiiat confirmation 
of suspicious findings is automatic because the information is available directly at the 
position of suspicion. Furthermore, if suspicious regions are offered by the secondaiy 
information (as in PET or CAD), tiien the viewer is drawn to the suspicious regions by their 
heightened visibility. 

In another exemplary embodiment of the invention, the secondary data can be data 
that is derived (computed from) either the primaiy modality image dataset and overplayed on 
the primaiy view. In this embodiment, an alignment (registration) process is not necessary 
when the secondary data is computed or derived from the primary image data. For instance, 
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for virtual colonoscopy applications, when viewing the colon wall, a region of the wall can be 
rendered using a translucent display to display the volume rendered CT data underneath the 
normal colon surface, to provide further context for evaluation. 

For instance, FIG. 6 is an exemplary view of a portion of a colon inner wall (60), 
wherein a primary view (61) is rendered having an overlay region (62) providing a 
translucent view of the CT image data below the colon wall within the region (62). In one 
exemplary embodiment, the translucent display (62) can be generated by applying a brightly 
colored color map with a low, constant opacity to the CT data and then volume rendering the 
CT data from the same viewpoint and direction as the primary image (61). 

In another exemplary embodiment, a translucent region (62) can be expanded to use 
the values of a second modality (e.g., PET) instead of just the CT data. This is helpful 
because the PET data can be mis-registered by several nun and be hidden under the normal 
sur&ce. This same technique can be used to ov^ay PET, SPECTT, CAD, shape, other 
modality data, or derived data onto the normal image. So, instead of viewing the CT data 
undemdith the colon sur&ce, one could view the secondary image data rendered below the 
colon sur&ce, in effect providing a window to peer into the second modality through the first 
modality. 

In anodier exemplary embodiment of tfie invention, secondary information may be 
derived data or tertiary information obtained from the results of automated segmentation, 
detection, diagnosis methods used to process the image information. This secondary 
information can be overlaid on a primary image to add context for user evaluation. FIG. 7 is 
an exemplary image of a colon wall displayed as a "filet" view (70) according to an 
exemplary embodiment of the invention. The exemplary filet view (70) is comprises a 
plurality of elongated strips (Sl-Sn) of similar width and lengdi, whmin each strip depicts a 
different region of a colon wall about a colon centerline for a gjven length of the imaged 
colon. The filet view (70) is a projecticm of the colon that stretches out the colon based on a 
colon centerline and is generated using a cylindrical projection about the centeriine. With 
this view, the portions of the colon that are curved are depicted as being straight such that the 
filet view (70) introduces significant distortion at areas of high curvature. However, an 
advantage of the filet view (70) is that a significantly lar^ portion of tiie colon sur&ce can be 
viewed in a single image. Some polyps may be behind folds or stretched out to look like 
folds, while some folds may be squeezed to look like polyps. 

The filet view (70) can be overiaid with secondary information. For instance, shape 
information such as curvature derived about the colon surface, and such shape information 
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can be processed to pseudo color the surface to distinguish various features, b the static 
filet view (70), it can be difficult to tell the difference between a depressed diverticula and an 
elevated polyp. To help differentiate polyps versus diverticula in the filet view (70) or other 
2D/3D projection view, methods can be applied to pseudo color depressed and elevated 
regions differently. In particular, in one exemplary embodiment, the shape of the colon 
surface can be computed and it can be determined at each such region to either color or 
highlight elevated regions and to color or de-enhance depressed regions. 

In another exemplary embodiment, the image data can be processed using automated 
diagnosis to detect potential polyps. The results of such automated diagnosis can be overlaid 
on the filet view of the image surfece (or other views) to highlight potential polyp locations. 

In other embodiment, highlighted PET data could be overlaid on top of the filet view 
(70) to indicated probable cancers. This overlay can be blended in and out with variable 
transparency. Data fiom modalities other tfian PET, such as SPECT or MRI, can also be 
overlaid and variable blended with the data, or laid out next to the CT data in alternating 
rows, for example. 

Although exemplary embodmients have been described herein with reference to the 
accompanying drawings, it is to be understood that the invention described herein is not 
limited to those precise embodiments, and that various other changes and modifications may 
be affected therein by one skilled in the art without departing from the scope or spirit of the 
invention. All such changes and modifications are intended to be included within the scope 
of the invention as defined by die appended claims. 
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What fa Claimed b; 

1 . A method for providing interactive navigation in a virtual image space, 

comprising: 

moving a virtual camera along a flight path in a virtual image space in response to 
user manipulation of a navigation control device; 

providing navigation assistance to the user by using the navigation control device to 
provide tactile feedback to the user upon die occurrence of a predefined event. 

2. The method of claim 1 , wherein providing navigation assistance user 
comprises im>viding force feedback to a steering control unit of the navigation control device 
to guide the user's flight patfi in a direction along a predetermined flight path. 

3. The metiiod of claim 2, wherein tfic predetermined flight path is a centerline 
through a lumen of a hollow organ. 

4. The method of claim 2, wherein the predefined event is based on a distance of 
the virtual camera fix)m the predetermined flight path. 

5. The method of claim 4, further comprising varying a magnitude of the force 
feedback applied to the steering control unit based on a measure of a distance of die virtual 
camera from the predetermined flight path. 

6. The method of claim 1 , wherein providing navigation assistance user 
comprises providing force feedback to a steering control unit of tfie navigation control device 
to guide the user's flight path in a directicMi away from an anatomical object to avoid collision 
with the object. 

7. The metiiod of claim 6, wherein the anatomical object is a virtual lumen inner 

wall. 

8. The method of claim 6, wherein the predefined event is based on a distance of 
the virtual camera to the anatomical object 
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9. The method of claim 8, further comprising varying a magnitude of the force 
feedback applied to the steering control unit based on a measure of the distance of the virtual 
camera to the anatomical object. 

10. The method of claim 6, further comprising providing force feedback to flight 
speed control unit of the navigation control device to reduce or stop the user's flight path to 
avoid collision with the anatomical object 

1 1 . The method of claim 1, wherein providing navigation assistance comprises 
providing force feedback to a flight speed control unit of the navigation control device to 
reduce a flight speed. 

12. The method of claim 1 1, wherein the predefined event is based on a distance 
of the virtual camera to an anatomical object in the virtual image space. 

13. The method of claim 1 1, wherein the predefined event is based on a tagged 
region of interest entering a field of view of the virtual camera. 

14. The method of claim 1 3, further comprising applying force feedback to a 
steering control unit to guide user's flight path in a direction toward the tagged region of 
interest 

1 5. The method of claim 13, further comprising providing a second form of tactile 
feedback to indicate the presence of the tagged region of interest within the field of view. 

16. A method for providing intmctive navigation in a vutual image 5pace» 
comprising: 

moving a virtual camera along a flight path at an actual flight speed in a virtual image 
space in response to user manipulation of a navigation control device; 

automatically modulating the actual flight speed upon the occurrence of a triggering 
event such that a perceived flight remains substantially constant. 
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1 7. The method of claim 16, wherein automatically modulating the actual flight 
speed is performed such that a perceived flight remains substantially similar to the actual 
flight speed before modulation. 

1 8. The method of claim 1 6, comprising: 

monitoring a position of the virtual camera in the virtual image space; and 
determining an occurrence of a triggering event when the flight path of the virtual 
camera becomes too close to an anatomical object in the virtual image space. 

19. The method of claim 18» wherein the virtual image space includes an organ* 
lumen and wherein the anatomical object comprises an inner lumen surfiice. 

20. The method of claim 1 6, comprising: 

monitoring a lumen width in a field of view of the virtual camera as the virtual camera 
travels along a centeriine path through a lumen of an virtual organ; and 

determining an occurrence of a triggering event when the lumen width is determined 
to fall outside a threshold range of lumen widths. 

21 . The method of claim 20, wherein automatically modulating die actual flight 
speed comprises gradually decreasing the flight speed as the lumen width decreases. 

22. The method of claim 20, wherein automatically modulating the actual flight 
speed comprises gradually increasing the flight speed as the lumen width increases. 

23. The method of claim 16, wherein the triggering event is based, in part, on a 
current actual flight speed. 

24. The method of claim 16, wherein automatically modulating the actual flight 
speed comprises overriding an input event generated by user operation of a flight speed 
control unit. 

25. The method of claim 16, wherein automatically modulating the actual flight 
speed comprises providing force feedback to a flight speed control unit operated by a user to 
automatically control the flight speed control unit. 
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26. A m^od for pioviding interactive navigation in a virtual image space» 
comprising: 

moving a virtual camera along a flight path at a flt^t speed in a virtual image space 
in response to user manipulation of a navigation control device; 

automatical^ overriding user control of the virtual camm and automatically 
controlling the flight path and flight speed upon tiie occurrence of a triggering event 

27. The method of claim 26, fuitber comprising automatically increasing a field of 
view (FOV) to aid the user in visualizing a region of Interest in the virtual image space. 

28. The method of claim 26, ccHnprising automatically modifymg a view direction 
to aid die user in visualizing a region of interest m the virtual space. 

29. An image data processing system, comprising: 

an image midering system for rendering multi-dmiensicnial views of an imaged object 
fit>m an image dataset of the imaged object; 

a gnvh"^ di^lay system fcH* diq>laying an unage of a rendered viw 
specified visualization parameters; and 

an intmctive navigation system which monitors a user's navigation throu^ a virtual 
image space of a displayed image and which provides user navigaticm assistance bi tfie form 
of tactile feedback by a navigation control unit operated by die usor, upon an occurrence of a 
predetemuned navigation event 
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Provide Specified Flight Path Through Virtual 
Space Associated with image Dataset 
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Process User input from a Navigation Control Device 
Operated by the User to Control Movement of a 
Virtual Camera along the Specified Flight Path or a User- 
Selected Flight Path Through the Virtual Space 
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Render and Display View of Image Object From the Viewpoint of 
the Virtual Camera in a Direction of the Specified Right Path or 
User-Selected Right Path 
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Automatically Generate Tactile Feedback to the User 
via the Navigation Control Device to provide Interactive 
Guided Navigation Assistance to User While 
Navigating through the Virtual Space 
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Navigate Along Flight Path Through Virtual 
Image Space at a Constant Flight Speed 




Automatically Modulate Actual Flight Speed 
Such That a User Perceivable Flight Speed is 
Similar to the Constant Flight Speed 
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Generate a Primary View of an Imaged Object 
Using Image Data of a First Imaging Modality 



Obtain Secondary Data Associated with 
Image Data of the Primary View 



Combine the Secondary Data with Associated Image 
Data in one or more Regions of Primary View 




Display Image of the Primary View such that those 
Regions of the Primary View having Combined Secondary 
Infomraation are Visibly Distinguishable 



n6.5 





SUBSTITUTE SHEET (RULE 26) 



wo 2006/042191 PCT/US2005/03d345 




62 



nG.6 



61 



60 





FI6.7 



SUBSTITUTE SHEET (RULE 26) 



